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INTRODUCTION 


Purpose 

The  DoD  Deputy  Direetor,  Operational  Test  and  Evaluation  /  Live  Fire  Testing 
(DDOT&E/LFT),  through  the  DOT&E,  tasked  Hicks  &  Associates  (HA&I)  to  conduct  a 
survey  of  models  and  simulations  (M&S)  being  used  in  support  of  defense  acquisition 
programs.  The  tasking  came  as  the  direct  result  of  issues  first  raised  in  an  article  which 
was  published  in  DSMC’s  Program  Manager  (PM)  Magazine  in  their  March  -  April  1999 
issue  entitled  “Meet  MASTER:  Modeling  &  Simulation  Test  &  Evaluation  Reform: 
Energizing  the  M&S  Support  Structure”.  The  MASTER  concept  includes  a  number  of 
proposals  including  the  formation  of  an  M&S  Consortium  and  the  possibility  of  funding 
this  Consortium  by  tapping  into  the  budgets  of  the  acquisition  systems  which  would 
benefit  from  M&S.  However,  when  the  question  of  how  much  is  currently  being  spent  in 
support  of  M&S,  and  related  questions  as  to  who  is  spending  it  and  for  what,  there  was 
little,  if  any,  comprehensive  data  upon  which  to  base  a  response.  The  purpose,  then  of 
the  Hicks  tasking  was  to  identify,  characterize,  and  determine  the  extent  of  M&S 
applications  across  a  range  of  program  office  activities  —  from  concept  exploration  and 
engineering  development  through  production  and  logistical  support  to  systems  in  the 
field. 

Objectives 

Key  objectives  of  the  survey  included  identifying  the  number  and  types  of  models  in  use, 
where  they  originated,  how  they  are  being  applied  in  support  of  the  acquisition  program, 
and  the  program  offices'  level  of  expenditure  on  M&S  tools.  In  addition,  the  survey 
sought  insights  into  how  M&S  activities  are  managed  within  program  offices,  including 
the  identification  of  potential  ‘best  practices’  and  problem  areas. 

Motivation 

The  project  was  motivated  by  several  factors.  First,  DoD  leadership  has  focused  over  the 
past  several  years  on  developing  a  High  Level  Architecture  (HLA),  and  related 
management  structures  such  as  the  Defense  Modeling  and  Simulation  Office  (DMSO),  to 
foster  simulation  interoperability  and  efficiencies  across  DoD  components.  These  efforts 
are  focused  largely  on  creating  the  technical  environment  and  infrastructure  necessary  to 
facilitate  future  M&S  activities.  In  contrast  to  this,  the  LFT&E  leadership  sought  to 
understand  more  about  how,  in  actual  practice,  models  and  simulations  are  actually 
viewed  and  utilized  from  the  Program  Managers’  perspectives;  not  theory,  not  policy,  but 
in  actual  day-to-day  application. 

New  DoD  policy  emphasis  on  Simulation-Based  Acquisition  (SBA)  is  a  second 
motivating  factor.  To  the  extent  DoD  leadership  plans  increasing  use  of  M&S  to 
substantially  reduce  the  time,  resources,  and  risks  associated  with  the  acquisition  process, 
a  better  understanding  of  how  M&S  is  currently  used  at  the  program  level  could  provide 
a  useful  point  of  departure.  Indeed,  despite  the  apparently  growing  role  of  M&S  in  a 
number  of  areas,  DoD  lacks  basic  information  on  the  extent  of  M&S  use  in  support  of 
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acquisition  program  offices.  DOT&E  therefore  sought,  through  this  survey,  to  improve 
the  general  level  of  understanding  of  how  M&S  tools  are  being  used  in  today's  program 
offices. 

Another  motivating  factor  was  cost.  An  initial  hypothesis  going  into  the  survey  was  that, 
despite  recent  efforts  to  focus  on  DoD-level  expenditures  supporting  HLA  and  attempts 
at  strengthening  M&S  oversight  in  the  Services,  most  M&S  expenditures  still  tend  to  be 
made  on  a  decentralized  basis  through  individual  program  offices.  Concern  was 
expressed  that  DoD  may  be  paying  more  than  once  for  the  same  models.  Further,  the 
issue  of  whether  existing  models  were  being  re-used  was  of  interest.  Sorting  out  program 
office  expenditures  for  M&S  was  thus  an  important  objective. 

A  final,  but  by  no  means  the  least  important,  motivation  for  this  survey  was  to  better 
inform  future  policies  involving  the  interrelationships  and  balanee  between  modeling  and 
simulation  and  more  traditional  forms  of  test  and  evaluation  (T&E)  —  a  complex  issue 
requiring  continuous  reassessment  by  the  operational  and  live-fire  test  communities. 


Projected  Results  and  Outcomes 

DOT&E  expects  this  survey  to  raise  cross-service  knowledge  of  M&S  is  used  at  the 
aequisition  program  level.  Where  the  survey  indieates  that  the  M&S  information  desired 
is  not  available,  the  report  may  suggest  the  need  for  further  visibility  into  M&S  activities 
and/or  improvements  in  routine  data  collection.  Anecdotal  data  reflecting  lessons  learned 
or  best  practices  should  be  of  use  to  program  managers.  Best  praetices  can  also  be  used 
for  benehmarking  future  M&S-related  policies  and  proeedures.  Where  eonclusions  and 
recommendations  are  drawn,  the  survey  may  contribute  to  an  improved  process  for 
access  to,  and  coordination  of,  M&S  information  and  provide  a  catalyst  to  stimulate  more 
effeetive  DoD  investments  in  M&S  support. 

As  the  survey  had  gathered  and  evaluated  sufficient  data  to  provide  preliminary  insights, 
a  series  of  briefings  were  initiated  to  the  DOT&E,  then  to  the  DMSO  office,  the  DDR&E, 
the  Service  Acquisition  Executives,  and  eventually  to  the  E1SD(AT&E).  As  a  result  of 
these  briefings,  tangible  positive  results  have  already  been  realized.  For  example,  the 
Undersecretary  of  Defense  for  Acquisition,  Technology  and  Eogistics  (USD(AT&E)) 
directed  his  staff  to  draft  changes  to  the  acquisition  instructions  in  response  to  insights 
highlighted  in  the  survey  results.  Furthermore,  the  Service  Acquisition  Executives  are 
co-sponsoring,  with  DOT&E,  a  series  of  Pilot  Projeets  that  provide  tangible,  real-life 
opportunities  for  M&S  to  better  support  the  acquisition  process. 

Overview 

This  report  begins  with  a  description  of  methodology  and  scope,  including  the 
organization  of  the  survey,  the  rationale  for  the  programs  selected  for  sampling,  access  to 
data,  and  methodological  challenges.  The  findings  are  then  deseribed  in  four  areas:  the 
eharacterization  of  model  types  and  uses;  M&S  management;  M&S  eosts;  and  model 
verification,  validation  and  accreditation  (VV&A)  activities.  The  positive  aspects  of 
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M&S  evolution  are  then  reviewed,  followed  by  current  issues  and  challenges.  The  report 
concludes  with  recommendations  in  two  categories:  areas  where  DOT&E  can  take 
unilateral  action;  and  recommended  actions  where  assistance  from  outside  organizations 
would  be  required. 

METHODOLOGY  AND  SCOPE 

Collecting  real-time  data  on  M&S  activities  supporting  DoD  acquisition  programs 
presented  several  challenges.  First,  modeling  and  simulation  tools  are  applied  to  a  wide 
range  of  activities,  with  the  number  of  tools  in  use  totaling  in  the  hundreds,  if  not 
thousands;  thus,  categorizing  M&S  activities  was  a  necessary  starting  point.  In  addition, 
with  over  200  acquisition  programs  underway,  selecting  a  small,  but  representative 
sample  of  programs  for  review  was  also  imperative.  After  a  series  of  discussions  with 
DDOT&E/LFT,  it  was  determined  that  a  survey  of  current  programs  in  the  acquisition 
process  would  be  the  most  viable  method  of  answering  the  questions  posed.  With  in 
excess  of  200  acquisition  programs  currently  underway  in  the  DoD,  it  was  clear  that  the 
survey  would  have  to  either  choose  to  do  a  superficial  evaluation  of  all  programs,  or  to 
select  a  representative  subset  of  programs  and  evaluate  them  in  depth.  The  latter 
approach  was  the  unanimous  choice.  It  was  decided  that  a  total  of  22  programs  would  be 
evaluated  which  would  cut  across  all  military  services,  air,  land  and  sea  platforms, 
weapons,  C4I  systems,  and  large  ACAT  ID  programs  as  well  as  smaller  programs  that 
were  at  the  beginning  or  nearing  the  end  of  their  development  cycle.  These  evaluations 
were  supplemented  by  interviews  of  program  office  and  other  Service  and  agency 
personnel  with  M&S  and/or  acquisition  experience.  A  list  of  personnel  interviewed  and 
the  documents  reviewed  during  a  literature  search  are  contained  in  Appendix  A. 

The  Survey 

The  categories  and  descriptions  of  M&S-related  activities  to  be  surveyed  were  drawn 
from  the  ""Taxonomy  of  M&S  Types  and  Applications  in  the  System  Acquisition  Process, 
December  1998.  ”  This  taxonomy  is  contained  in  ‘A  Roadmap  for  Simulation  Based 
Acquisition,  ”  produced  by  the  Joint  Simulation  Based  Acquisition  Task  Force  under  the 
auspices  of  the  Acquisition  Council,  DoD  Executive  Council  for  Modeling  and 
Simulation  (EXCIMS).  Though  still  regarded  as  a  work  in  progress,  this  taxonomy 
reflected  the  most  recent  and  detailed  effort  of  the  M&S  community  to  categorize  a  large 
number  of  distinct  models  and  tools.  A  list  of  specific  questions  was  developed,  based  on 
requirements  for  data  identified  by  the  sponsor.  These  categories  and  questions  provided 
a  uniform  framework  for  the  survey.  The  collection  of  basic  M&S-related  data  was 
emphasized  over  analysis  of  the  models  themselves  or  their  effectiveness.  (The  survey 
framework,  however,  did  provide  a  limited  opportunity  for  subjective  or  anecdotal  inputs 
on  the  quality  and  utility  of  the  M&S  tools  in  use.)  In  addition,  the  survey  sought 
descriptions  of  how  M&S  activities  are  managed  within  the  program  office.  The  survey 
template,  questionnaire,  and  taxonomy  are  contained  in  Appendix  B. 

Twenty-two  programs  were  identified  for  the  survey  including  representative  programs 
from  the  Services  and  a  mixture  of  platforms,  weapons,  and  C41SR  systems.  The 
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programs  selected  also  varied  in  size,  age  and  technical  complexity,  providing  a  broad 
cross-section  of  acquisition  programs. 

Project  goals  and  methodology  were  briefed  to  the  Service  acquisition  chains  of 
command  beginning  with  Service  Acquisition  Executives  (SAEs)  and  their  staffs, 
including  the  Program  Executive  Officers  (PEOs),  and  the  Program  Managers  or  their 
representatives  in  the  Program  Offices.  The  program  offices  were  given  30  days  to 
respond  (most  received  extensions).  They  utilized  a  mixture  of  internal,  contractor,  and 
other  government  support  to  develop  their  response.  Of  the  twenty-two  programs 
selected  for  the  survey,  twenty-one  programs  responded  and  cooperated  with  the  study 
team.  The  only  program  initially  selected  which  did  not  participate,  the  Joint  Strike 
Tighter  (JSE)  program,  did  not  due  so  primarily  due  to  the  proprietary  nature  of  the 
competitive  development  that  was  ongoing  between  the  two  major  teams,  thereby  making 
the  discussion  and  dissemination  of  this  information  problematic.  The  twenty-one 
programs  are  listed  in  Figure  1 . 


Figure  1 :  Summary  of  Survey  Responses  to  Date 


Program 

Component 

System 

Type 

Current  StatusF 

FRP  Date* 

Crusader 

Armv 

Platform 

MSII  decision  2001 

1QFY06 

Comanche 

Armv 

Platform 

MSli  decision  FY02 

1QFY07 

M1A2  Unarade 

Armv 

Platform 

FRP 

3QFY94 

ATACMS  BIkl  I/BAT 

Armv 

Weapoh 

LRIP 

3QFY00 

Javelin 

Armv 

Weapon 

FRP 

3QFY97 

SADARM 

Armv 

Weapon 

LRIP 

4QFY98 

FAAD  C2 

Armv 

C4I3R 

FRP 

3QFY95 

C2  Vehicle 

Armv 

C4ISR 

LRIP 

1QFY00 

F/A-1B  E/F 

N  arvv 

Platform 

LRIP 

3QFY00 

V-22  Osprev 

N  avv/U  SMC 

Platform 

LRIP 

2QFY00 

LPD-17 

N  aivv/U  SMC 

Platform 

EMD 

3QFY07 

AIM-9X 

N  arvv 

Weapon 

LRIP 

1QFY02 

AN/BSY-2  (SSN-21) 

N  arvy 

C4I3R 

Sea  Trials  on  SSN-22 
(U3S  CONNECTICUT) 

N/A 

UHF  Follcw-On 

Navv 

C4ISR 

Completino  FRP 

4QFY88 

SLAM-ER 

N  arvv 

Weapon 

FRP 

2QFY99 

F-22 

USAF 

Platform 

LRIP 

3QFY03 

B-2 

USAF 

Platform 

IOC 

N/A  (did  not 
enter  FRP) 

EELV 

USAF 

Platform 

MSII  Decision  FY99 

2QFY03  (MSIII 
decision 
1QFY03) 

ABL 

USAF 

Weapon 

MSII  decision  FY03 

2QFY05 

SBIRS 

USAF 

C4I3R 

MSII  decision  1996 

First  GEO  sat 
deliv.  FY02; 
HEO  FY03; 
LEO  FY-04 

SFWP3I 

USAF 

Weapon 

FRP 

3QFY96 

*Source:  DOT&E  FY98  Annual  Report  to  Congress 
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Program  Type 
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Program  office  responses  were  provided  directly  to  H&AI.  The  results  outlined  below 
refleet  program  offiee  responses  to  the  formal  survey  supplemented  with  informal 
comments  and  insights  related  to  M&S  from  PEOs  and  others  in  the  Serviee  aequisition 
chains  and  the  Offiee  of  the  Seeretary  of  Defense  (OSD).  Data  extraeted  during  the 
literature  seareh  are  also  integrated  into  this  report. 

Limitations 

It  is  important  to  note  several  limitations  in  attempting  to  draw  conelusions  and 
generalizations  from  the  programs  surveyed.  First,  while  they  represent  a  eross-seetion 
of  programs,  the  programs  surveyed  also  cover  some  twenty-five  years  or  more  in 
program  age.  During  this  period,  significant  changes  in  M&S,  and  other  information 
teehnologies,  as  well  as  numerous  changes  in  aequisition  polieies  and  proeedures  have 
affeeted  program  offiees.  Under  these  eireumstances,  efforts  to  eompare  M&S  activities 
among  programs  must  be  earefully  considered. 

The  availability  of  M&S-related  data  appears  to  have  been  affected  by  a  number  of 
variables,  ineluding  program  age,  reeord  keeping  proeedures,  staff  eontinuity  and 
knowledge  of  program  history.  In  addition,  the  methods  used  within  eaeh  program  to 
complete  the  survey  also  varied,  as  well  as  the  background  and  experience  of  those  who 
eompleted  the  survey.  In  some  eases,  for  example,  programs  offiees  have  personnel 
dedieated  full  time  to  coordination  and  oversight  of  M&S  aetivities;  in  other  programs, 
the  point  of  contaet  for  the  survey  had  only  limited  M&S  experienee.  Similarly,  M&S 
experts  may  have  been  unfamiliar  with  aeeounting  and  budgeting  proeedures  and/or  how 
to  gain  further  insights  into  M&S  costs. 

The  survey  involved  a  matrix  of  M&S  eategories  and  questions  providing  a  total  of  168 
possible  data  cells  to  be  filled-in  by  the  program  offiees.  Many  program  offices  appeared 
to  use  a  distributed  approaeh  to  the  survey,  dividing  the  work  among  experts  from  the 
program  offiee,  eontraetors,  and  Serviee  laboratories  and  FFRDCs.  Programs  also  eaeh 
set  their  own  limits  (e.g.  80  hours)  on  how  mueh  time  would  be  devoted  to  this  effort. 
This  proeess  of  distributed  data  entry  inherently  produces  data  of  uneven  quality. 
Variations  in  personnel  background  and  expertise,  combined  with  limitations  of  time  and 
availability  of  data,  also  contributed  in  some  cases  to  empty  data  cells,  ineomplete  entries 
or  entries  not  comparable  aeross  the  sixteen  program  matrices. 

One  of  the  most  important  eaveats  about  the  data  is  that  the  survey  results  represent  only 
a  point  in  time.  It  is  not  possible  to  use  the  data  to  draw  eonclusions  about  trends  or 
ehanges  in  M&S  development  and  use. 

Despite  sueh  limitations,  the  survey  produeed  objeetive  and  anecdotal  data  of  potential 
use  to  program  managers,  the  M&S  eommunity,  and  aequisition  polieymakers  in  general. 
As  an  initial  effort,  it  highlights  what  is  known  and  unknown  about  M&S  activities  at  the 
program  level;  and  with  a  modest  level  of  analysis  suggests  potential  opportunities  for 
improvement  and  identifies  issues  for  further  investigation. 
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FINDINGS  AND  CONCLUSIONS 


Overview 

The  wide  variety  of  programs  selected  for  the  survey  was  a  major  plus,  resulting  in  a 
broad  and  representative  spectrum  of  inputs.  However,  each  program  is  a  product  of 
specific  circumstances  involving  different  acquisition  strategies,  acquisition  policies, 
relationships  to  other  programs  (as  in  follow-ons),  funding  profiles,  and  whether  the 
program  is  in  early  development  or  in  post-production  support. 

The  survey  represents  a  limited  sample  of  acquisition  programs  at  a  specific  point  in  time 
from  which  conclusions  cannot  easily  be  drawn.  The  conclusions  and  recommendations 
outlined  below  should  be  considered  in  this  context. 

Characterization  of  Model  Types  and  Uses 

This  section  of  the  report  characterizes  the  types  of  M&S  that  are  reported  in  use  by  the 
responding  programs.  Such  a  characterization  is  of  interest  since  it  provides  insight  into 
potential  opportunities  for  developing  common  models  or  community-standards  and  the 
types  of  expertise  that  may  be  needed  for  their  development. 

Even  though  the  taxonomy  used  for  the  survey  is  generally  accepted  within  the  M&S 
community,  there  is  some  overlap  among  model  types.  Further,  the  category  definitions 
are  quite  detailed  but  still  are  subject  to  interpretation.  As  a  result,  program  offices 
reported  the  same  model  under  different  categories.  Every  attempt  was  made  to 
adjudicate  these  variations  during  analysis  of  the  data.  Figure  2  shows  the  distribution  of 
M&S  reported  by  the  program  offices  by  model  category. 

Figure  2:  M&S  Characterization 
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The  Engineering  Design/Manufaeturing  (ED/M)  and  Combat  M&S  categories, 
respectively,  comprise  the  most  M&S.  ED/M  M&S  include  Virtual  3-D  Eayouts,  Virtual 
Manufacturing  Plant  Simulations,  and  Constructive  Engineering  and  Manufacturing 
Process  Models.  The  predominance  of  Combat  and  ED/M  M&S  is  not  surprising 
considering  that  acquisition  program  offices  focus  on  system  design  and  development 
rather  than  activities  associated  with  later  phases  of  the  system  life  cycle,  such  as  training 
and  logistics. 

Engagement  M&S  constituted  the  large  percentage  of  Combat  M&S.  Several  weapons 
systems  and  platform  programs  reported  using  more  than  one  engagement  model.  Closer 
examination  of  this  data  revealed  that  the  need  for  different  engagement  models  with  a 
program  probably  arose  from  needing  different  levels  of  fidelity  for  different  purposes 
and  the  need  to  address  system  performance  across  a  variety  of  threats.  M&S  in  both  the 
engagement  and  threat  categories  tended  to  rely  on  physics-based  modeling. 

The  next  largest  category  of  Combat  M&S  was  Virtual  models  which  included  Hardware 
in  the  Eoop  (HWIL),  Man  in  the  Eoop  (MITE),  and  virtual  system  simulations.  Survey 
results  indicated  that  most  of  these  simulations  were  being  used  to  support  testing  and/or 
training.  Of  special  note,  the  AIM-9X,  ATACMS  BAT,  and  SLAM-ER  program  inputs 
described  using  the  HWIL  facilities  to  develop  pretest  predictions  and  to  facilitate  post¬ 
test  analysis.  The  linking  of  HWIL  to  MITE  capabilities  represents  an  opportunity  to 
support  testing  and  training  in  a  “system  of  systems”  context  using  M&S.  Eor  example, 
the  SLAM-ER  program  intends  to  link  its  HWIL  to  the  E/A-I8  MITE  simulation  so  that  a 
pilot  can  practice  flying  a  weapon  from  a  platform  to  a  target.  HWIL  capabilities  have 
benefited  from  the  investment  of  corporate  capital  and  the  leveraging  of  that  investment 
by  government  customers  building  on-site  government  HWIL  facilities. 

The  number  of  environment  models  may  seem  unusually  low.  However,  representations 
of  environments  such  as  terrain,  weather,  and  oceans  are  included  integrally  in  many  of 
the  Combat  M&S.  Also,  the  data  showed  that  several  programs  are  now  using  the  some 
of  the  same  Environment  M&S  (e.g.,  the  LOWTRAN  model  for  atmosphere).  It  is 
possible  that  over  time,  several  M&S  have  been  incorporated  into  or  reduced  to  a  single 
model. 

Platform  programs  used  the  most  M&S.  This  observation  appeals  to  common  sense  since 
ships,  aircraft,  and  tanks  require  complex  integration  of  diverse  subsystems,  interaction 
with  multiple  environments,  face  multiple  threats,  and  entail  many  man/machine  interface 
issues.  Eor  the  same  reasons,  platform  programs  used  relatively  more  combat  and 
logistics  models  than  weapons  and  C4ISR  programs. 

C4ISR  programs  used  a  relatively  higher  percentage  of  training  M&S  than  DID  platform 
and  weapons  programs.  This  result  may  be  due  to  the  common  practice  of  evolving  a 
C4ISR  system  from  its  virtual  representation  in  simulation  to  the  actual  functional 
capability  with  training  occurring  throughout  the  simulation  evolution.  Eor  instance,  the 
M&S  suite  for  the  EAAD  C2  program  steps  from  a  simulation  of  end-to-end  system 
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functionality  to  a  simulation  which  evaluates  system  effectiveness  to  a  training  and 
exercise  eapability  that  ean  be  used  in  the  field  or  in  garrison. 

If,  rather  than  sorting  by  program  type,  the  data  are  sorted  by  program  maturity,  it  is 
evident  that  the  types  of  M&S  currently  in  use  by  a  program  offiee  depends  on  where  the 
program  is  in  the  aequisition  proeess.  For  example,  the  M1A2  Upgrade  program  (FRP 
1994)  is  using  many  more  test,  training,  and  logisties  simulations  than  the  LPD-17  (FRP 
2007)  program  that  is  using  primarily  engineering  and  combat  M&S. 

Over  25  M&S  were  used  by  more  than  one  Program.  The  eommonality  was  strongest 
within  program  types,  e.g.,  weapons  systems  using  LOWTRAN.  Also  some  prime 
contractors  used  the  same  model  aeross  different  programs. 

Use  of  eommon  models  ean  be  a  positive  faetor  for  the  aequisition  community  since  it 
presents  an  opportunity  for  a  eommon  basis  of  understanding  and  evaluation.  It  also 
affords  an  opportunity  for  cost  sharing  among  programs.  However,  it  is  desirable  for  the 
commonly  used  models  to  be  the  best  model  for  a  particular  purpose,  i.e.,  to  insure  that 
RELEX  is,  indeed  the  best  reliability  model  for  the  Programs  that  are  using  it.  Eor 
example,  the  1999  Defense  Science  Board  (DSB)  study  on  T&E  noted  that  many  of  the 
models  in  use  today  were  developed  more  than  twenty  years  ago  when  computing 
technology  limited  algorithmic  approaches  and  the  threats  and  system  eapabilities  were 
mueh  different.  Another  key  question  to  eonsider  eoncerning  common  model  use  is 
whether  the  model  has  been  eertified  for  its  intended  use  and  whether  the  intended  use  is 
within  the  model  assumptions. 


Conclusion:  Requirements  for  M&S  vary  by  program  type  and  program  maturity. 
Availability  of  a  eommon  set  of  M&S  eould  benefit  program  offices  by  allowing  them  to 
conserve  sehedule  time  and  funding. 


M&S  Management 

According  to  the  Defense  Systems  Management  College  (DSMC)  report.  Simulation 
Based  Acquisition:  A  New  Approach  (December  1998),  SBA  “is  an  iterative,  integrated 
produet  and  process  approaeh  to  aequisition,  using  modeling  and  simulation,  that  enables 
the  warfighting,  resouree  alloeation,  and  aequisition  communities  to  fulfill  the 
warfighter’s  materiel  needs,  while  maintaining  Cost  As  an  Independent  Variable  (CAIV) 
over  the  system’s  entire  life  cyele  and  within  the  DoD’s  system  of  systems.” 

The  DSMC  report  goes  on  to  say  that  there  are  five  essential  features,  or  enablers,  of  an 
SBA  proeess  that  need  to  be  implemented  by  the  aequisition  community.  Eirst,  “early 
user  involvement  is  essential  in  defining,  refining,  and  balaneing  requirements.  This  is 
the  eritieal  first  step  in  produeing  better,  faster,  and  eheaper  material  solutions.”  Second, 
a  synthetic  environment  allows  an  integrated  produet  and  process  design  (IPPD)  team  to 
“coneurrently  explore  greater  numbers  of  possible  material  solutions  than  is  possible 
within  the  eurrent  aequisition  process.  Third,  the  iterative  nature  of  the  SBA  design 
proeess  will  enable  IPPD  teams  to  converge  systematically  on  optimal  solutions  more 
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efficiently  than  is  currently  possible.  Fourth,  SB  A  will  support  changing  the  role  of 
testing  into  that  of  being  an  integral  part  of  the  design  process.  Finally,  SB  A  supports 
informed  tradeoff  analyses  through  a  Decision  Risk  Analysis  process.” 

The  DSMC  report  then  provides  detail  about  the  activities  that  constitute  each  of  these 
five  enablers.  These  activities  include  (but  are  not  limited  to)  development  of  an  M&S 
Support  Plan,  creation  of  a  Modeling,  Simulation  and  Analysis  (MS&A)  staff  that 
includes  users,  preparation  and  implementation  of  a  W&A  plan/process,  re-use  of  M&S 
across  acquisition  program  phases  and  across  programs  to  include  the  use  of  Commercial 
Off-the-Shelf  (COTS)  tools,  development  of  a  collaborative  environment,  and 
incentivizing  the  performance  of  the  M&S  developers  and  users.  Although 
implementation  of  these  activities  alone  is  not  sufficient  for  SBA  success,  they  are 
deemed  necessary  in  the  DSMC  report.  Figure  3  summarizes  the  SBA  precepts  and  these 
enabling  activities. 

Figure  3:  M&S  Management  Requirements  for  SBA,  Acquisition  Reform 


The  survey  questions  asked  about  the  status  of  each  of  these  activities  within  the  Program 
Offices.  Data  concerning  M&S  management  provide  insight  into  the  current  state  of 
implementation  of  the  SBA  vision.  Depending  on  one’s  perspective,  the  resulting 
answers,  summarized  in  Figure  4,  are  either  good  news  or  bad  news  for  the  status  of  SBA 
implementation.  Across  the  board,  roughly  half  of  the  surveyed  programs  reported 
conducting  the  activities  that  are  considered  necessary  to  SBA  success.  Not  surprisingly, 
the  newer  programs  reported  engaging  in  more  of  these  activities  than  the  older  or  legacy 


11 


programs.  Implementation  of  these  aetivities  varied  some  by  Service.  For  instance,  the 
Army  programs  reported  a  higher  number  of  these  activities  as  ongoing  than  the  other 
two  Services. 


Figure  4;  Incorporation  of  Selected  M&S  Management  Activities 
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•  Services  vary  in  their  approach  to  M&S  management 

•  Approach  also  varies  by  program  maturity 


Many  of  the  programs  have  developed  M&S  Support  Plans  that  provide  detailed 
information  about  the  M&S  that  are  being  used  within  the  program.  However,  these 
plans  do  not  usually  address  requirements  for  future  M&S  growth  nor  identify  the 
funding  necessary  to  accomplish  those  improvements. 

A  former  Director  of  the  Defense  Modeling  and  Simulation  Office  noted  that  having 
M&S  experts  within  the  staff  of  a  Program  Office  is  critically  important  to  the 
appropriate  and  successful  development  and  use  of  M&S.  A  few  of  the  programs 
reported  the  existence  of  M&S  staffs  and  others  noted  that  they  had  formed  M&S  IPTs 
that  included  members  of  the  user  community.  These  IPTs  were  responsible  for 
determining  how  M&S  would  be  applied,  as  well  as  defining  requirements  for  future 
M&S  development. 

Despite  the  existence  of  many  Validation,  Verification  and  Accreditation  (VV&A)  policy 
documents,  such  as  DoD  Instruction  5000.61,  “DoD  Modeling  and  Simulation 
Verification,  Validation,  and  Accreditation,”  few  of  the  programs  had  implemented 
overarching  VV&A  processes,  as  shown  in  Figure  4.  Many  programs  had  VV&A’d  one 
or  more  of  their  M&S.  W&A  is  discussed  in  more  detail  later  in  this  report. 
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Figure  4  shows  re-use  of  M&S  across  acquisition  phases  of  programs  and  across 
acquisition  programs.  M&S  re-use  across  a  program’s  acquisition  phase  may  have  been 
underreported  since,  as  noted  earlier  in  this  report,  survey  respondents  tended  to  be 
focused  on  the  “here  and  now.”  Newer  programs,  such  as  LPD-17,  noted  their  intent  to 
mature  and  use  the  same  core  set  of  M&S  throughout  the  system  lifecycle.  Legacy 
programs,  such  as  M1A2,  used  models  originally  developed  by  their  predecessor 
programs,  e.g.,  the  MlAl.  M&S  re-use  across  acquisition  programs  also  occurred  across 
programs  that  had  the  same  prime  contractor,  e.g.,  the  F-15  and  the  F-18E/F.  Although 
the  re-use  columns  of  Figure  4  indicate  that  the  majority  of  the  programs  in  the  survey 
were  re-using  models,  less  than  20%  of  the  reported  M&S  were  used  by  multiple 
programs  in  the  survey. 

Another  instance  of  model  re-use  occurs  when  a  program  uses  COTS  packages.  Over 
75%  of  the  programs  reported  using  COTS  M&S.  In  particular,  COTS  tools  that 
supported  engineering  design  and/or  manufacturing,  such  as  Pro  E,  CATIA,  and 
Unigraphics  were  prevalent.  As  with  the  more  program-specific  cases  of  model  re-use, 
COTS  can  provide  a  common  framework  for  insight  and  analysis. 

As  Eigure  4  shows,  to  date,  collaborative  environments  have  been  implemented  by  less 
than  30%  of  the  program  offices  surveyed.  The  survey  results  indicated  that  even  the 
terminology,  “collaborative  environment,”  is  not  well  understood.  Rather  than 
interpreting  collaborative  environment  as  an  overarching  architecture  that  facilitates  the 
linkage  of  M&S  across  distributed  locations  and  organizations,  to  several  program  offices 
the  term  connoted  a  collegial  work  environment  in  which  IPTs  shared  information.  Eor 
the  few  programs  that  have  implemented  or  begun  to  implement  meaningful  collaborative 
environments  there  were  some  encouraging  success  stories.  Eor  instance,  the  Javelin 
Program  had  started  the  evolution  towards  a  collaborative  environment  by  developing  an 
Integrated  Test  and  Simulation  Network.  This  network  connects  the  program  office, 
prime  contractor,  the  Army  Missile  Command  Research  Development  and  Engineering 
Center,  the  Redstone  Technical  Test  Center,  the  Javelin  user,  and  the  Army  Infantry 
School  Training  and  Doctrine  Command.  This  collaborative  environment  gave  the 
program  timely  access  to  the  expertise  required  for  engineering  design  and  testing.  It  will 
also  support  remote  training. 

Another  example  of  a  collaborative  environment  is  found  within  the  EPD-17  program. 
One  of  the  newest  programs  in  the  survey,  LPD-17  implemented  an  Integrated  Product 
Data  Environment  based  on  “tailored  applications  of  COTS  products  and  commercially- 
available  foundation  systems  (CAD  and  PDM).”  All  personnel  involved  in  any  way  in 
the  EPD-17  Program  can  use  the  environment  to  access  and  work  with  program 
management,  requirements,  and  engineering  design  information  and  models.  The  State 
of  Eouisiana  provided  the  funding  for  LPD-17’s  collaborative  environment. 

Einally,  as  reflected  in  Eigure  4,  the  survey  results  showed  that  slightly  more  than  half  the 
programs  included  M&S  activities  in  their  prime  contractor’s  Statement  of  Work.  This 
observation  may  explain  why  several  program  offices  indicated  that  M&S  development 
costs  were  not  separable  from  investments  in  other  program  activities  —  if  M&S  are  not 
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addressed  as  separate  tasks  under  the  SOW,  it  is  likely  to  be  difficult  for  the  government 
to  track  the  associated  costs.  Only  one  program  office,  Crusader,  reported  incentivizing 
the  contractor  for  M&S-related  efforts. 


Another  M&S  management  issue  of  interest  to  the  study  sponsor  was  the  development 
and  ownership  of  M&S.  The  survey  results  showed  that  industry  is  the  predominant 
developer  of  the  M&S  used  by  the  program  offices.  In  addition,  industry  retains 
ownership  of  nearly  40%  of  the  M&S  included  in  the  survey.  Figure  5  summarizes  M&S 
developers  and  owners  based  on  survey  data. 


Figure  5;  Developers  &  Owners 
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Thirteen  of  the  twenty-one  programs  participating  in  the  survey  supplied  data  about  who 
developed  the  models.  Contractors  developed  nearly  half  of  the  M&S  described  by  these 
13  program  offices.  Over  75%  of  these  M&S,  i.e.,  more  than  60  M&S,  were  developed 
by  the  program’s  prime  contractor  or  one  of  its  team  members.  For  example,  Boeing  or 
Bell  developed  several  M&S  for  the  V-22  program,  Hughes  developed  M&S  for  the  UHF 
Follow-On  and  Lockheed  Martin  developed  M&S  for  the  F-22  program.  Other 
contractors  developing  M&S  for  the  programs  included  the  Aerospace  Corporation, 
Aerojet,  Logistics  Management  Institute,  RAND,  and  SAIC.  Small  companies,  software 
development  companies  and  universities  accounted  for  the  remaining  development. 

Twenty-three  percent  of  M&S  were  developed  by  an  organization  within  the  program’s 
sponsoring  Service.  The  Army  developed  the  highest  percentage  of  these  models,  using 
the  Army  Research  Lab,  Army  Materiel  Systems  Analysis  Activity,  components  of  the 
Training  and  Doctrine  Command  (TRADOC),  Space  and  Missile  Defense  Command 
(SMDC),  and  U.S.  Forces  Command  (FORSCOM).  The  Navy  followed  with  the  next 
highest  percentage  of  Service-developed  M&S  relying  on  the  Naval  Undersea  Warfare 
Center,  Naval  Surface  Weapons  Center,  and  NAVAIR.  The  survey  results  did  not 
indicate  whether  any  of  these  Service  organizations  used  industry  support  in  developing 
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the  models.  Other  government  organizations  also  contributed  to  the  development  of 
M&S  for  the  Programs.  These  entities  included  the  Department  of  Energy’s  National 
Labs,  National  Institute  for  Standards  and  Technology  (NIST),  and  National  Aeronautics 
and  Space  Agency  (NASA). 

Ownership  data  were  provided  for  over  300  M&S.  These  data  show  that  the  contractor 
who  developed  and/or  is  using  the  assets  owns  the  largest  percentage  (38%)  of  M&S. 
This  38%  included  ownership  of  COTS  tools  where  ownership  means  having  a  license 
for  the  package.  Contractor  ownership  may  become  an  issue  when  M&S  are  considered 
proprietary.  One  of  the  highest  numbers  of  contractor-owned,  proprietary  models  on  a 
single  program  occurred  with  the  Space-Based  Infrared  Systems  (SBIRS)  and  the 
Aerospace  Corporation.  In  this  case,  the  contractor  is  working  with  the  government 
providing  architectural  insight  and  information  on  design  trades.  It  may  be  that  the 
nature  of  the  govemment/contractor  relationship  warrants  the  proprietary  nature  of  the 
models. 


Conclusion:  Some  of  the  management  activities  that  are  considered  necessary  to  full 
realization  of  the  SBA  vision  are  being  implemented.  These  activities  are  particularly  in 
evidence  on  the  new  acquisition  programs  in  the  survey.  However,  implementation  of 
these  activities  alone  is  not  a  sufficient  condition  for  SBA  to  succeed. 


Industry  plays  the  key  role  in  development  of  M&S,  although  the  extent  of  this  role  does 
vary  somewhat  by  Service.  Industry  also  retains  ownership  of  a  large  percentage  of 
M&S.  This  fact  is  worth  some  consideration  in  the  event  that  a  proprietary  classification 
prevents  visibility  into  the  M&S  or  negatively  impacts  the  ability  to  link  models  for 
“system  of  system  ”  evaluation  purposes. 

Validation,  Verification  and  Accreditation  Efforts 

Validation,  Verification,  and  Accreditation  (VV&A)  efforts  appear  to  be  challenging,  at 
least  based  on  survey  results.  The  extent  of  VV&A  activities  varied  significantly  across 
the  surveyed  programs.  In  addition,  the  technical  approach  and  expertise  involved  in 
program  office  VV&A  efforts  varied. 

Referring  to  the  DSMC  Report,  “Simulation  Based  Acquisition:  A  New  Approach  ”  and 
the  DoD  M&S  Glossary,  validation  is  the  process  of  determining  the  degree  to  which  a 
model  or  simulation  is  an  accurate  representation  of  the  real  world  from  the  perspective 
of  the  intended  uses  of  the  model  or  simulation.  Verification  is  the  process  of 
determining  that  a  model  or  simulation  implementation  accurately  represents  the 
developer’s  conceptual  description  and  specification.  Verification  also  evaluates  the 
extent  to  which  the  model  or  simulation  has  been  developed  using  sound  and  established 
software-engineering  techniques.  Accreditation  is  the  official  certification  that  a  model 
or  simulation  is  acceptable  for  use  for  a  specific  purpose. 

Implementation  of  these  three  processes  is  intended  to  provide  a  decision-maker  with 
confidence  in  the  model  results.  Figure  6  summarizes  the  status  of  W&A  efforts  among 
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the  programs  in  the  survey.  The  growing  role  of  M&S  in  aequisition  and  the  potential  of 
VV&A  to  inerease  a  deeision-maker’s  eonfidenee  make  it  somewhat  surprising  that  the 
VV&A  status  of  35%  of  the  M&S  was  unknown  or  at  least  was  not  available  for  the 
survey.  This  result  leads  one  to  question  whether  the  model  users  know  and  understand 
the  limits  and  frailties  of  the  models  and  whether  those  limits  are  taken  into  account 
when  basing  engineering  design  or  operational  decisions  on  the  model  outcomes. 
Similar  questions  arise  when  considering  the  VV&A  status  of  the  COTS  packages  used 
by  the  Programs.  The  VV&A  “status”  ascribed  to  most  COTS  packages  was  “not 
applicable.”  This  status  is  an  acceptable  one  when  the  COTS  user  knows  the  pedigree  of 
the  package  including  its  strengths,  weaknesses,  and  limits.  It  is  also  acceptable  when  the 
COTS  package  is  used  to  facilitate  some  other  M&S  function,  e.g.  visualization  of  data  or 
storage/retrieval  of  results. 

Figure  6:  VV&A  Overview 


■  Supfilving  Contractor 

■  Pro^amSt^tPT 

■  Sponsoring  Serv  ice 

■  In  Process 

□  Joint  Process 

□  COTS 

B  H  on  plica  bletJ  nknovirvTtot 
Prouided 


•  Uncertainty  about  ‘pedigree'"  of  M&S  being  used  (3S%) 

•  Potential  conflicts  of  interest  (14%) 

•  VV&A  sta  ndards  for  COTS  M&S? 

•  Use  of  joint/independent  processes  low  (7%) 


Fourteen  percent  of  the  VV&A  efforts  reported  depended  on  the  involvement  or 
leadership  of  the  contractor  who  developed  the  model.  Depending  on  the  procedures  that 
are  implemented,  errors  may  be  more  easily  overlooked  when  contractors  VV&A  their 
own  models.  An  alternative  view  is  that  the  contractor  developing  the  system  (and  the 
supporting  M&S)  has  a  vested  interest  in  insuring  W&A  is  conducted  rigorously  —  the 
contractor  wants  the  system  to  perform  as  required. 

VV&A  efforts  conducted  or  underway  were  performed  by  Program  Staff  or  IPT  or  by  a 
specified  organization  within  the  Program’s  sponsoring  Service  on  over  25%  of  the 
models.  Programs  such  as  the  C2  Vehicle,  ATACMS  BAT,  and  V-22  indicated  that  they 
were  working  with  Service  organizations  with  expertise  in  VV&A,  e.g.  Army  Materiel 
Systems  Analysis  Activity  (AMSAA)  for  Army  Programs,  and  were  using  an  iterative 
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process  of  comparing  developmental  test  results  to  model  predictions.  Further,  some 
programs  such  as  LPD-17  reported  use  of  the  DMSO  Reeommended  VV&A  Practices 
Guide. 

For  about  7%  of  the  models,  joint  organizations,  without  a  vested  interest  in  the  program, 
were  condueting  the  VV&A.  For  example,  F/A-18E/F  and  SLAM-ER  were  putting  at 
least  some  of  their  models  under  the  scrutiny  of  the  Joint  Teehnical  Coordinating  Groups 
on  Aircraft  Survivability  and  Munitions  Effectiveness.  Such  groups  have  access  to 
domain  and  subject  matter  experts  and  typically  have  a  breadth  of  experience  across 
platform  or  weapons  systems  acquisition  programs.  In  addition,  the  E-22  Program  noted 
that  it  was  using  several  models  considered  part  of  the  Air  Eorce  Standard  Analysis 
Toolkit  and  certified  by  the  Joint  Accreditation  Support  Activity  (JASA)  and/or  the  Joint 
Teehnical  Coordinating  Group  on  Aircraft  Survivability.  As  the  1999  DSB  on  T&E 
notes,  JASA,  which  also  supported  the  W&A  efforts  of  the  P/A-18  and  AIM-9X 
programs,  has  significant  expertise  in  VV&A.  The  AE  Standard  Analysis  Toolkit  with 
models  accredited  by  JASA  may  be  a  significant  resource  for  Program  Managers  since 
they  do  not  have  to  pay  for  model  development  or  provide  the  funds  for  VV&A.  One 
eoncern  is  that  the  M&S  in  this  toolkit  may  be  used  beyond  the  situations  for  which  they 
were  accredited. 

The  most  pervasive  theme  concerning  VV&A  from  the  survey  data  is  that  the  term 
“VV&A”  has  a  plethora  of  different  meanings.  While  some  good  news  eoncerning 
VV&A  was  provided  in  the  preceding  paragraph,  e.g.  the  use  of  joint,  independent 
experts,  many  models  are  being  accepted  without  VV&A.  Threat  models  seemed  to  pose 
a  partieular  challenge.  The  most  common  W&A  approach  for  threat  models  was  to 
simply  compare  with  historical  intelligence  documents. 

The  1999  T&E  DSB  notes  that  funding  and  expertise  needed  for  VV&A  efforts  may  be 
difficult  to  obtain.  Eurther,  the  Program  Manager  is  usually  under  severe  schedule 
pressure  so  a  slip  in  model  availability  due  to  VV&A  activities  may  not  be  considered 
aceeptable.  Einally,  although  there  is  no  shortage  of  VV&A  policy,  i.e.  OSD  and  each 
Service  have  VV&A  policies,  implementation  is  intended  to  be  “flexible”.  In  other 
words,  the  policy  documents  may  not  provide  the  Program  Manager  with  the  information 
necessary  to  structure  a  solid  and  standardized  W&A  approaeh.  Undoubtedly,  all  of 
these  circumstances  contribute  to  the  range  of  VV&A  approaches  found  in  the  survey. 


Conclusion:  VV&A  policies  abound  but  the  nature  and  depth  of  W&A  activities  varies 
widely  across  programs.  Availability  of  expertise  and  cost  and  schedule  pressures  may 
curtail  a  program  office’s  intended  VV&A  praetices. 
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Costs 


An  important  goal  of  the  survey  was  understanding  the  costs  associated  with  M&S 
activities.  Better  understanding  of  M&S  costs  supports  several  management  objectives. 
These  objectives  include  providing  DoD  leadership  with  insight  into  funds  that  are  being 
spent;  assessing  the  adequacy  of  current  M&S  funding  in  light  of  other  DoD  priorities; 
promoting  more  effective  and  efficient  M&S  investments;  and  highlighting  potential 
tradeoffs  between  M&S  investments  and  those  for  other  T&E  methods  and/or 
infrastructure. 

The  survey  confirmed  the  current  lack  of  management  visibility  into  program 
expenditures  for  M&S  activities.  Figure  7  indicates  the  availability  of  cost  data  on  M&S 
activities  within  the  program  offices  participating  in  the  survey.  Significant  reasons  for 
the  lack  of  visibility  include; 

•  Standard  cost  accounting  procedures  that  do  not  provide  for  segregation,  reporting  or 
tracking  of  M&S  costs. 

•  M&S  costs  that  are  not  included  as  a  reportable  DoD  Element  of  Expense. 

•  Contracts  that  do  not  track  M&S  activities  unless  a  specific  model  or  tool  is  listed  as  a 
deliverable  item. 

•  The  costs  of  multipurpose  M&S  tools  in  use  by  contractors  that  are  reflected  in  other 
costs  to  the  government,  such  as  engineering  support  to  the  program  or  delivery  of 
end  items. 

•  Most  programs  are  not  required  to  track,  and  therefore  are  not  currently  tracking, 
program  expenditures  specifically  for  M&S  activities. 

Figure  7:  M&S  Cost  Overview 


□  Cost  Unknown  (Developed 
Outside  Program) 

□  Cost  Unknown  (Proprietary  to 
Contractor) 

■  Costs  Unknown  (Not 
Separable) 

■  No  Information  Provided 


■  Progs  w/Data  for  100%  of  M&S 

■  Progs  w/Data  far >40%  of  M&S 
□  Progs  w/Data  far  <25%  of  M&S 


•  M&S  development  and  application  costs  data  are  not  readily  available 
within  acquisition  programs 
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Only  10  of  the  21  Programs  supplied  eost  data  on  M&S  development.  Colleetively,  these 
10  Programs  supplied  eost  data  on  30%  of  the  M&S  reported.  Only  four  of  the  ten  eost- 
reporting  Programs  supplied  cost  data  on  all  of  the  M&S  in  use  by  the  program.  It  is 
impossible  to  generalize  the  results  displayed  on  the  accompanying  graphic  to  a 
meaningful  statement  about  the  level  of  investment  in  M&S  other  than  setting  a  lower 
bound  and  also  the  fact  that  the  data  are  readily  available.  Even  for  the  four  programs 
that  did  provide  complete  cost  information,  it  is  not  possible  to  draw  conclusions.  The 
four  programs  are  a  mix  of  new  (LPD-17)  and  mature  (Javelin)  and  legacy  or  follow-on 
efforts  (AN/BSY-2,  ATACMS-BAT).  Since  the  programs  are  in  different  phases  of  the 
acquisition  process  they  are  likely  at  different  points  in  their  eventual  expenditure  of 
M&S  funds.  The  Programs  also  represent  platform,  weapons,  and  C4ISR  program  types 
that,  as  shown  previously,  vary  in  the  number  and  type  of  M&S  used. 

Figure  8;  M&S  Cost  Summary 


Program 

Approx. 

Acquisition  Cost 

M&S  D  evelopment 

E  xpendituresto 
Date 

Reported  M&S  exp.  to 
date  as  %  of  Total 
Acquisition  Cost 

%  of  M&S  with 
Cost  Data 

LPD-17 

$10B 

$38  M 

<1% 

100% 

ATACMS/B  AT 

$5B 

$25.2M 

<1% 

100% 

Javelin 

$4B 

$48  M 

1.2% 

100% 

AHjBSY-2 

$3B 

$58.3M 

1.9% 

100% 

SADARM 

$3B 

$14.6M 

<1% 

78% 

V-22 

$37B 

$50.2M 

<1% 

44% 

FAAD  C2 

$1B 

$37.6M 

3.7% 

40% 

SLAM-ER 

$.5B 

$8.1M 

1.6% 

21% 

F/A-18E/F 

$46B 

$15.1M 

<1% 

5% 

SB  IRS 

$8B 

$28  M 

<1% 

<1% 

M&S  its  Percentage  of  RDT&E: 
The  ATACMS/BAT  Example 


RDT&E  Expenditures 
toDate* 

M&S  Investmmt 
to  Date* 

M&S  Investment  as  % 
of  RDT&E 

Expoidtiresto  Date* 

$1.46B 

$25.2M 

1.7% 

TbroK^  PY99 


Viewing  M&S  cost  as  a  percentage  of  total  acquisition  cost  may  be  misleading  as  well.  It 
is  unlikely  that  funds  will  be  spent  on  M&S  evenly  over  the  entire  acquisition  lifecycle  of 
the  program.  It  is  more  typical  that  M&S  funds  will  be  required  during  the  RDT&E 
phase  of  the  acquisition  process  as  opposed  to  the  procurement  phase.  Hence  as  an 
example,  the  M&S  investment  as  a  percentage  of  RDT&E  expenditures  to  date  was 
calculated. 
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However,  the  survey  was  able  to  identify  several  sources  of  M&S  support  to  the  program 
offices  which  suggest  a  list  of  activities  to  be  considered  in  developing  an  estimate  of 
total  DoD  M&S  expenditures.  Examples: 

•  Contractor  Support.  Survey  results  suggested  that  contractors  develop  and  utilize 
M&S  tools  from  basically  four  sources  in  their  support  of  acquisition  programs: 

•  Generic  COTS  tools; 

•  Contractor-proprietary  tools  developed  internal  to  the  company,  funded  through 
Independent  Research  and  Development  (IR&D),  with  applicability  across 
multiple  commercial  and/or  military  programs; 

•  Generic  or  proprietary  tools  further  modified  by  the  contractor  to  fit  particular 
program  needs; 

•  Custom  tools  developed  (or  purchased  from  others)  by  the  contractor  for  a 
specific  program  purpose  or  tailored  application. 

•  Lab  or  FFRDC  Support.  Service  laboratories  and  product  centers,  along  with  several 
Federally  Funded  Research  and  Development  Centers  (FFRDC),  develop  and 
maintain  models  with  broad  applicability  to  Service  acquisition  programs.  Typically, 
the  initial  development  and  routine  maintenance  of  these  models  are  funded  within 
the  laboratory  or  center  budget.  Acquisition  programs  pay  for  the  incremental  cost  of 
operating  the  model,  and/or  any  necessary  modifications  to  the  model,  in  support  of 
their  specific  program  needs.  In  these  cases,  the  cost  of  such  a  model  to  the  program 
should  be  reflected  in  a  billing  and  reimbursement  record  between  the  Lab  and  the 
System  Program  Office  (SPO). 

•  Service  or  DoD  M&S  Repositories.  In  some  cases,  the  Services  and  DoD  have 
created  central  repositories  of  M&S  tools  above  the  Lab  level.  Typically,  models 
called  out  for  centralized  control  at  the  Service  level  are  those  with  Tri-Service-wide 
applicability  across  commands  or  missions.  One  example  would  be  more  complex, 
dynamic,  high-end  models  involving  the  integration  of  combat,  logistics,  and  threat 
data  in  a  force-on-force  scenario.  As  in  the  case  of  Lab  support,  individual  program 
offices  do  not  pay  for  development  or  maintenance  of  such  models,  but  are  required, 
to  reimburse  the  owner  for  support  provided  to  the  program.  In  some  cases,  a 
program  may  need  to  subsidize  directly  Lab  or  Service  level  development  of  a  large 
and  complex  model  to  simulate  an  environment  or  operational  scenario  necessary  to 
evaluate  program  effectiveness. 

In  addition  to  identifying  sources  of  M&S  support  to  program  offices,  other 
methodologies  were  sought  to  isolate  the  cost  of  M&S  activities  at  the  program  level. 
The  challenge  was  posed  in  the  following  question:  Where  and/or  how  would  a  Program 
Manager  find  an  audit  trail  leading  to  government  and  contractor  expenditures  on  M&S 
in  support  of  a  given  program?  After  discussion  with  several  experts,  including  a  leading 
M&S  expert  at  the  Defense  Systems  Management  College,  a  former  Director  of  the 
Defense  Contract  Management  Command,  and  a  Service  Auditor  General,  it  was 


20 


concluded  that  program  costs  and  expenditures  assoeiated  with  M&S  aetivity  eould  not 
be  estimated  without  a  detailed  review  of  a  program's  work  breakdown  strueture. 


Conelusions:  At  present,  data  eoneeming  the  investment  in  M&S  at  the  program  level  is 
not  available  to  an  extent  that  supports  poliey  formulation  and  resouree  alloeation.  Some 
opportunities  and  proeedures  are  available  for  future  use  should  M&S  eost  traeking  be 
recognized  as  desirable. 


SUMMARY 

•  Charaeterization  of  Model  Types  and  Uses:  Requirements  for  M&S  vary  by 
program  type  and  program  maturity.  Availability  of  a  eommon  set  of  M&S 
eould  benefit  program  offiees  by  allowing  them  to  eonserve  sehedule  time 
and  funding. 

•  M&S  Management:  Some  of  the  management  activities  that  are  eonsidered 
neeessary  to  full  realization  of  the  SBA  vision  are  being  implemented. 
These  aetivities  are  particularly  in  evidence  on  the  new  aequisition  programs 
in  the  survey.  However,  implementation  of  these  activities  alone  is  not  a 
suffieient  eondition  for  SBA  to  succeed. 

•  Validation,  Verifieation  and  Aeoreditation  Efforts:  VV&A  polieies  abound 
but  the  nature  and  depth  of  VV&A  activities  varies  widely  across  programs. 
Availability  of  expertise,  eost  and  sehedule  pressures  may  curtail  a  program 
office’s  intended  VV&A  praetiees. 

•  Costs:  At  present,  data  eoneeming  the  investment  in  M&S  at  the  program 
level  is  not  available  to  an  extent  that  supports  poliey  formulation  and 
resouree  alloeation.  Some  opportunities  and  proeedures  are  available  for 
future  use  should  M&S  eost  traeking  be  reeognized  as  desirable. 


RECOMMENDATIONS 

The  following  major  reeommendations  are  based  on  the  survey  data,  literature  seareh  and 

interviews.  These  recommendations  were  presented  to  the  Underseeretary  of  Defense 

(Aequisition,  Teehnology  and  Logisties)  in  Mareh  2000. 

•  Program  offiees  should  develop  a  formal  plan  early  in  the  program  and  get  buy-in  to 
the  plan  by  the  aequisition  deeision-makers  and  testing  eommunity. 

•  Program  offiees  should  use  M&S  to  aid  in  test  design  and  make  pre-test  predietions. 
Test  results  should  then  be  used  to  validate  and  improve  models  and  simulations. 

•  Program  offiees  should  speedy  that  the  models  and  simulations  used  in  the 
aequisition  proeess  are  deliverables  under  the  aequisition  contract.  The  program 
offiee  should  implement  measures  to  insure  suffieient  visibility  and  eonfidenee  in  the 
results  of  models  and  simulations. 
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•  OSD  and  the  Service  Acquisition  Executives  should  develop  improved  techniques  to 
guide  the  use  of  M&S  in  the  proposal  development  and  proposal  evaluation  process. 

•  OSD  should  assume  a  leadership  role  in  determining  how  to  best  use  M&S  to  explore 
interoperability  issues.  Particular  emphasis  should  be  placed  on  the  potential  of 
hardware-in- the-loop  and  man-in-the-loop  simulations. 

At  the  request  of  the  USD  (AT&L),  these  insights  coming  from  this  survey  were 
incorporated  in  the  draft  revision  of  DoD  5000. 2-R  (Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  and  Major  Automated  Information  System  Acquisition 
Programs).  These  recommendations  are  discussed  in  more  detail  in  the  following 
paragraphs. 

The  role  and  value  of  formal  Modeling  and  Simulation  Plans 

Of  the  twenty-one  programs  surveyed,  eleven  programs  had  developed  M&S  Support 
Plans.  These  plans  are  useful  in  providing  an  overview  of  the  M&S  tools  that  are  being 
used  by  the  acquisition  programs  but  do  not  define  the  future  requirements  for  acquisition 
program  support.  In  other  words,  the  plans  do  not  define  the  technical  capability  needed 
or  funding  requirements  for  M&S  over  the  course  of  the  acquisition  program.  Lack  of  a 
focus  on  M&S  requirements  means  that  the  Program  Manager  may  not  have  the  M&S 
tools  (or  the  funding  and  technical  expertise)  available  when  needed,  for  example,  to 
support  T&E. 

A  requirements-focused  M&S  plan  enables  the  Program  Manager  to  better  justify  and 
protect  the  necessary  funding.  The  1999  Defense  Science  Board  on  T&E  noted  that  “the 
sizeable  costs  associated  with  the  development  and  employment  of  M&S  capabilities  can 
be  daunting  to  the  Program  Manager  who  requires  such  capabilities  as  an  adjunct  to 
physical  testing  of  a  weapon  system.  ”  As  the  DSB  report  shows,  M&S  development  and 
usage  costs  are  not  well  understood  even  after  the  investment  has  been  made.  Inclusion 
of  funding  projections  in  M&S  plans  may  engender  improved  assessment  of  costs  and 
benefits  and,  certainly,  a  vision  of  what  kinds  of  M&S  will  be  needed  at  what  points 
during  the  acquisition  process. 

Developing  a  requirements-based  M&S  plan  would  also  facilitate  improved 
responsiveness  from  the  Science  and  Technology  (S&T)  community.  The  survey  results 
indicated  that  the  acquisition  community  was  not  using  the  latest  “Best-of-Breed”  M&S 
tools  developed  by  the  S&T  organizations.  Eor  example,  outdated  versions  of 
hydrodynamics  codes  were  still  in  use  by  some  programs  although  the  S&T  community 
has  greatly  improved  upon  these  codes.  Defining  M&S  requirements  near  the  start  of  an 
acquisition  program  would  allow  the  S&T  community  to  better  apply  its  resources. 

Another  benefit  of  a  forward-looking  M&S  plan  is  that  it  facilitates  M&S  re-use  in  time 
across  acquisition  phases  and  across  acquisition  programs.  The  survey  results  indicated 
that  M&S  re-use  was  occurring  to  a  limited  degree  across  acquisition  phases  on  15  of  the 
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21  programs.  13  programs  were  using  M&S  that  had  been  used  by  other  aequisition 
programs.  Often  the  prime  eontractor  drove  this  re-use,  i.e.,  a  eontractor  used  the  same 
M&S  on  multiple  programs.  Sharing  of  requirement-driven  plans  among  Program 
Managers  eould  increase  opportunities  for  M&S  re-use. 

Development  of  these  plans  will  undoubtedly  require  more  reliance  on  M&S  expertise. 
Interviews  conducted  during  the  M&S  survey  indicated  that  PMs  who  have  dedicated 
M&S  staffs  or  access  to  M&S  Integrated  Product  Teams  (IPTs)  may  realize  a  higher 
return  on  investment  on  M&S  support.  Five  of  the  21  programs  in  the  survey  had  M&S 
staffs  or  IPTs. 

Use  of  M&S  for  pre-test  predictions;  use  test  results  to  validate  and  improve  M&S 

Programs  under  the  statutory  purview  of  Live  Fire  Test  and  Evaluation  are  required  to 
perform  pre-test  predictions.  For  example,  A1M-9X,  Javelin,  and  ATACMS  BAT  are  all 
using  M&S  for  pre-test  predictions  and  using  test  data  to  improve  the  M&S.  While  the 
LFT&E  statutory  requirements  apply  to  systems  representing  over  80%  of  all  DoD 
procurement  dollars,  there  are  some  systems  that  do  not  require  EET&E.  Similarly,  there 
are  also  many  non-MDAP  programs  that  have  no  statutory  lOT&E  requirement  either. 
While  the  EET&E  program  has  enforced  the  discipline  of  pretest  predictions  being 
provided  prior  to  every  EET&E  test  since  its  inception,  this  has  not  been  true  of  other 
types  of  T&E.  Eurther,  the  1999  DSB  on  T&E  notes  that  “verification,  validation  and 
accreditation  (VV&A)  as  presently  practiced  with  respect  to  M&S  techniques  in  general 
is  not  sufficiently  disciplined  to  inspire  confidence  in  their  use  in  the  T&E  process.” 

There  is  no  shortage  of  VV&A  guidelines  in  the  Services  and  DoD  organizations.  The 
survey  results,  however,  indicated  that  implementation  varies  widely.  Eor  instance,  the 
survey  results  documented  approaches  to  M&S  validation  such  as  comparison  of  model 
results  with  threat  documents  or  historical  data,  comparison  of  model  results  with  test 
data,  and  review  of  M&S  output  by  IPTs.  Data  presented  earlier  in  this  report  show  that 
in  many  cases  the  M&S  pedigree  was  unknown  or  uncertain  (35%). 

Views  on  what  these  results  imply  for  the  credibility  of  M&S  in  acquisition  are  disparate. 
Some  acquisition  executives  believe  the  benefits  are  outweighed  by  the  cost  and  schedule 
implications  of  VV&A.  Others  see  a  rigorous  W&A  regimen  as  essential  prior  to 
reliance  on  M&S-generated  results.  Requiring  use  of  M&S  for  pre-test  predictions  and 
use  of  test  results  to  validate  M&S  is  thought  by  many  to  be  an  “80%  solution”  towards 
improving  the  credibility  and  utility  of  M&S  support  without  overburdening  program 
schedules  and  resources. 

In  addition  to  requiring  pre-test  predictions,  another  logical  step  is  to  explore 
opportunities  to  better  use  M&S  as  a  tool  in  test  design.  The  survey  results  indicated  that 
few  programs  were  using  M&S  in  this  way.  As  the  Deputy  Undersecretary  of  Defense 
for  Science  and  Technology  (DUSD(S&T)),  points  out,  M&S  resources  like  the  High 
Performance  Computing  Network  are  applicable  and  available  to  help  PMs  evaluate 
alternative  test  designs. 
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Value  of  making  Models  &  Simulations  deliverables  under  the  acquisition  contract 

Establishing  M&S  as  deliverables  to  the  government  gives  the  PM  staffs  an  opportunity 
for  insight  into  engineering,  design,  and  performance  trade-offs  that  is  not  necessarily 
available  currently.  It  also  provides  the  test  community  with  a  basis  for  determining 
when  M&S  results  can  be  accepted  in  support,  of  or  in  place  of,  tests.  Without  the  full 
disclosure  and  access  facilitated  by  the  contract  deliverable  process,  it  is  generally 
impossible  for  the  government  to  base  management  decisions  or  test  and  evaluation 
adequacy  on  the  results  of  M&S  alone. 

A  case  in  point  for  this  new  approach  is  the  Evolved  Expendable  Eaunch  Vehicle  (EEEV) 
Program.  The  EEEV  Program  Office  has  contractually  defined  the  “insight”  concept  by 
requiring  that  the  contractors  provide  the  government  access  to  ah  of  the  tools  they 
employ  to  design,  develop,  and  test  their  systems.  The  Program  Office  engineers  assess 
the  adequacy  of  the  contractors’  methods,  the  assumptions  which  go  into  their  models, 
how  and  when  the  models  are  applied,  and  how  the  data  or  results  from  the  models  are 
used  for  design  and  development  decision  making.  This  approach  has  allowed  the  PM  to 
gain  confidence  in  each  contractor’s  development  activity. 

As  mentioned  earlier  in  this  report,  industry  is  the  predominant  force  in  M&S 
development,  use,  maintenance  and  ownership.  Over  half  of  the  359  M&S  in  the  survey 
were  developed  by  industry  and  industry  retained  ownership  of  nearly  40%  of  the  359 
M&S.  The  majority  of  these  models  are  viewed  as  “proprietary.”  To  link  M&S  and  to 
use  them  across  programs  to  support  “system  of  systems”  evaluations  will  require  that  the 
government  have  additional  insight  into  the  models  and  the  supporting  assumptions  and 
architectures.  Similarly,  using  M&S  to  support  T&E  objectives  means  that  the  Service 
Operational  Test  Agencies  (OTAs)  need  access  to  the  M&S  and  supporting  data. 
Requiring  M&S  deliverables  is  one  step  towards  gaining  this  insight. 

Another  advantage  of  specifying  M&S  as  contractual  deliverables  is  that  it  provides  a 
better  basis  for  tracking  the  cost  of  M&S.  Opinions  on  whether  or  not  tracking  the  cost 
of  M&S  is  useful  are  polarized.  Many  M&S  experts  believe  that  tracking  M&S  costs  is 
not  meaningful  since  M&S  is  an  integral  tool  in  the  acquisition  process.  Others  believe 
that  some  knowledge  of  funds  invested  in  M&S  is  valuable  in  justifying  and  protecting 
M&S  investments.  An  assessment  of  the  benefits  of  M&S  including  costs  avoided,  time 
saved,  risks  reduced,  and  system  cost/performance  trade-offs  made  more  effectively 
should  accompany  tracking  of  costs. 

The  cost  tracking  inherent  in  specifying  M&S  deliverables  may  also  benefit  industry. 
The  survey  results  showed  that  many  contractors  were  investing  corporate  Internal 
Research  and  Development  (IR&D)  funds  in  developing  M&S  including  significant 
investments  in  Hardware-in-the-Eoop  (HWIL)  and  Man-in-the-Eoop  (MITE)  capabilities. 
Also  the  survey  results  showed  that  PMs  were  able  to  make  extensive,  cost  effective  use 
of  these  HWIL/MITE  simulations.  However,  as  profit  margins  and  returns  on  investment 
in  DoD-related  activities  spiral  downward,  corporate  Chief  Einancial  Officers  (CEOs)  are 
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reluctant  to  condone  such  investments  of  scarce  IR&D  funds.  Improved  cost  tracking 
through  the  contractual  deliverable  process  may  allow  companies  to  realize  a  better  return 
on  investment  in  these  expenditures  of  corporate  IR&D. 

Requiring  that  M&S  become  deliverables  has  other  implications  for  the  contracting 
community.  As  shown  previously  in  this  report,  only  11  of  the  21  programs  in  the  survey 
included  M&S  activities  in  the  contractor’s  Statement  of  Work.  Of  those  11  programs, 
only  one  incentivized  the  contractor  for  performance  on  M&S  related  tasks.  In 
specifying  M&S  as  deliverables,  the  PMs  should  consider  increased  use  of  an  incentive 
structure  for  contractually  responsive  M&S  delivered  on  time  and  within  budget. 

Clearly,  contractors  will  be  reluctant  to  make  certain  models  and  simulations  deliverables 
for  fear  that  the  intellectual  property  and  competitive  advantage  inherent  in  these  models 
will  be  subject  to  dissemination  to  competitors.  This  situation  will  obviously  require 
extreme  care  in  contracting.  It  may  even  require  the  development,  at  government 
expense,  of  industry-wide  M&S  that  are  distinct  from  the  proprietary  design  M&S  used 
for  competitive  advantage.  Moreover,  where  to  draw  the  line  in  terms  of  making  M&S 
deliverables  needs  to  be  assessed.  Questions  such  as  “are  only  newly  developed  M&S 
required  to  be  deliverables”,  and  “is  the  contractor  support  of  the  M&S  as  well  as  the 
M&S  themselves  required  to  be  deliverables”,  will  have  to  be  answered. 

The  Role  of  Models  and  Simulation  in  the  Proposal  Process 

The  proponents  of  Simulation-Based  Acquisition  (SBA)  argue  that  M&S  should  be 
introduced  into  the  acquisition  process  as  early  as  possible  and  used  in  an  iterative 
manner  throughout  the  course  of  the  acquisition  program.  In  the  context  of  this  goal, 
envisioning  the  contractor’s  models  and  simulations  as  substantive  parts  of  the  proposal, 
and  envisioning  the  government’s  models  and  simulations  as  substantive  proposal 
evaluation  tools  is  logical.  The  contractor  and  government  need  to  specify  those  M&S 
tools  at  the  start  of  the  acquisition  process,  perhaps  even  in  the  Request  For  Proposal 
(RFP). 

Such  specification  is  likely  to  be  a  useful  first  step  towards  accomplishing  the  still  elusive 
SBA  vision.  M&S  used  in  evaluating  proposals  are  likely  to  be  used  by  contractors  in 
developing  the  proposals.  Then,  having  made  the  investment  in  particular  M&S  for  the 
proposal,  the  contractor  is  likely  to  continue  to  use  them  at  least  in  the  early  stages  of 
system  design  and  development. 

There  is  a  need  to  identify  the  M&S  that  will  be  used  for  evaluation  purposes.  Even  in  a 
relatively  small  sample  of  21  programs,  359  M&S  tools  were  in  use.  Of  these  359,  over 
100  were  categorized  as  Combat  M&S,  including  campaign,  mission,  engagement,  and 
models,  that  are  used  extensively  in  the  early  stages  of  system  acquisition.  This  large 
collection  of  M&S  represents  an  even  larger  collection  of  underlying  assumptions,  input 
data,  and  computational  techniques. 
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In  addition  to  specifying  the  technical  M&S  that  will  be  used  in  proposal  evaluation,  the 
government  should  specify  the  cost  or  affordability  models  that  will  also  be  used. 
Despite  the  DoD’s  emphasis  on  Cost  as  an  Independent  Variable  (CAIV),  the  survey 
results  revealed  that  few  M&S  were  being  used  to  support  CAIV  analysis. 

The  use  of  Models  and  Simulation  to  explore  interoperability  issues 

Interoperability  is  one  of  the  major  stated  goals  in  the  DoD  today.  It  is  a  difficult  goal  to 
pursue  and  for  which  to  quantify  success.  In  some  cases,  M&S  may  offer  one  of  the  few 
viable  means  for  cost  effectively  and  time  efficiently  analyzing  and  testing 
interoperability.  The  M&S  survey  results  indicate  that  few  programs  were  using  M&S  or 
had  plans  to  use  M&S  to  evaluate  the  interoperability  of  that  program  with  other 
programs  in  acquisition.  In  fact,  the  Space-Based  Infrared  System  (SBIRS)  was  the  only 
program  of  twenty-one  that  reported  modeling  in  support  of  joint  systems 
interoperability. 

Some  M&S  tools  offer  extraordinary  potential  to  help  realize  the  vision  of 
interoperability.  For  instance,  HWIL  and  MITL  simulations  are  an  emerging  part  of  the 
T&E  infrastructure  that  can  prove  invaluable  in  this  context.  These  capabilities  support 
repeatable  test  and  training  events  among  system  or  mission  area  components  in  a  cost 
effective  and  schedule  efficient  manner.  A  case  in  point  reported  in  the  survey  is  the 
integration  of  the  SLAM-ER  HWIL  simulation  with  the  E/A-18  MITL  simulator  that 
enabled  a  pilot  to  train  and  analyze  delivering  a  weapon  to  its  target.  This  example 
portrays  the  most  elemental  level  of  interoperability  testing  and  training  since  the  SLAM- 
ER  and  L/A-18  were  designed  to  satisfy  well-specified  interface  requirements  relative  to 
one  another.  However,  this  first  step  is  indicative  of  the  opportunity  to  examine  “system 
of  systems”  interactions  by  linking  multiple  HWIL  and  MITL  capabilities. 

Another  tool  that  may  be  useful  in  supporting  interoperability  objectives  is  a 
collaborative  environment,  i.e.,  an  overarching  architecture  that  facilitates  the  linkage  of 
M&S  across  distributed  locations  and  organizations.  The  survey  results  discussed  above 
indicate  that  less  than  30%  of  the  programs  had  established  collaborative  environments 
and  that  a  significant  number  of  the  respondents  were  even  unfamiliar  with  that  term. 

Interoperability  may  also  require  use  of  common  M&S  across  programs  to  ensure  a 
consistent  frame  of  reference.  The  survey  results  showed  that  of  the  359  models  and 
simulations  in  use  across  21  programs,  at  least  twenty-five  M&S  were  being  used  by 
more  than  one  program.  If  these  M&S  were  to  be  used  across  programs  to  support 
interoperability  activities,  several  questions  would  need  to  be  answered.  These  questions 
include  whether  these  M&S  are  “best  of  breed,”  what  are  the  strengths  and  weaknesses  of 
the  M&S,  and  what  are  the  limits  on  extension  and  application  of  the  M&S.  The  Air 
Lorce  Standard  Analysis  Toolkit,  a  collection  of  M&S  that  have  been  accredited  for  use 
for  specific  purposes,  exemplifies  an  approach  that  might  be  useful  for  interoperability 
needs. 
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NEXT  STEPS 


A  version  of  DoD  5000. 2-R  that  incorporates  the  general  recommendations  discussed 
above  is  now  in  coordination.  In  addition,  USD(AT&L)  and  DOT&E  have  agreed  that 
M&S  plans  will  be  more  thoroughly  examined  during  program  reviews  and  during 
reviews  of  Test  and  Evaluation  Master  Plans  (TEMPs)  and  other  test  plans.  Eurther, 
DOT&E  plans  to  work  with  DE1SD(S&T)  to  develop  M&S  for  use  in  operational  test 
design  and  to  improve  migration  of  M&S  applications  from  the  S&T  community  to 
program  offices. 

Discussions  of  the  survey  results  between  DOT&E  and  the  three  Service  Acquisition 
Executives,  produced  the  idea  for  a  series  of  Pilot  Projects.  The  Pilot  Projects  will 
examine  the  opportunities  for  using  M&S  to  effectively  support  the  acquisition  process. 
Each  Service  has  nominated  one  program  for  consideration.  The  projects  involve  (1) 
characterizing  the  data  needed  to  support  acquisition  decision  making,  (2)  categorizing 
the  potential  opportunities  for  M&S  support  in  generating  these  data,  and  (3)  describing 
the  risks  and  benefits  of  using  M&S-generated  data  in  support  of  specific  acquisition 
decisions.  The  end  result  of  the  Pilot  Projects  will  hopefully  be  a  methodology  that  can 
be  used  to  guide  the  development  and  application  of  M&S  to  support  system  acquisition. 

The  dynamic  field  of  M&S  has  the  potential  to  improve  the  acquisition  process.  Eor 
some  programs,  particularly  those  requiring  system  of  systems  engineering,  M&S  tools 
may  offer  a  practical  means  of  examining  and  exercising  all  parts  of  the  system 
repeatedly  in  an  integrated  manner.  To  make  better  use  of  M&S,  PMs  need  improved 
planning,  funding,  contractual,  and  education/training  support.  The  changes  discussed 
above  should  help  to  expand  and  improve  the  use  of  M&S  in  acquisition  and  help  the 
DoD  continue  to  evolve  towards  its  goal  of  SBA. 
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Appendix  A 


SURVEY  SUPPLEMENT 
Survey  Supplement 

A  literature  review  and  interviews  supplemented  the  survey  results  with  some  key  M&S 
experts.  A  representative  list  follows. 

Documents  reviewed: 

DSB  Task  Force  on  Simulation  for  Innovation  (1998) 

Joint  Simulation  Based  Acquisition  Task  Force  Report  for  the  DoD  EXCIMS  (1998) 
DDR&E  Acquisition  Task  Eorce  on  M&S  (1994) 

DTSE&E  Study  on  Effectiveness  of  M&S  in  Weapon  System  Acquisition  (1996) 

DSMC  System  Acquisition  Managers  Guide  for  the  Use  of  M&S  (1994) 

Simulation  Based  Acquisition:  A  New  Approach  (DSMC  1998) 

NRC  1998:  Statistics,  Testing  and  Defense  Acquisition 
Program  simulation  support  plans  (SSPs) 

The  Study  on  Effectiveness  of  M&S  in  Weapon  System  Acquisition  completed  in  1996 
for  the  Director  of  Developmental  Test,  System  Engineering,  and  Evaluation  (DTSE&E) 
was  the  most  similar  in  purpose  and  scope  to  this  effort,  however,  it  provided  primarily 
anecdotal  data  on  the  application  of  M&S  within  a  few  programs.  In  addition,  the  cost 
data  discussed  were  in  terms  of  expected  cost  savings  from  use  of  models  in  place  of  tests 
and  so  were  not  consistent  with  the  intent  of  this  effort. 

The  Simulation  Support  Plans  provided  by  the  Program  Offices  were  an  invaluable 
source  of  information  in  that  they  generally  contained  more  detailed  descriptions  of  the 
M&S  than  the  survey  inputs  and  provided  information  about  how  the  M&S  were  used  in 
conjunction  with  one  another  and  with  other  program  activities  such  as  T&E.  An  area  of 
concern  is  that  these  plans  did  not  contain  a  requirements  roadmap  detailing  how  the 
M&S  will  be  matured  and  describing  the  funding  necessary  to  do  that. 

Additional  interviews: 

DMSO  (Col  Kenneth  Konwin,  et  al) 

Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Eogistics  and  Technology) 
(Ms.  Ellen  Purdy) 

SAE/XOC  (Dr.  Jacqueline  Henningsen,  et  al) 

DSMC  (Mr.  Randy  Zittel) 

DOT&E/IDA  (Dr.  David  Sparrow) 

The  interviews  with  recognized  M&S  experts  provided  a  much  broader 
perspective  of  the  evolution  of  M&S  than  the  survey  data  did.  In  particular,  the 
discussions  with  Defense  Modeling  and  Simulation  Office  (DMSO)  and  Army  Secretariat 
for  Research,  Development,  and  Acquisition  (SARDA)  personnel  were  invaluable  in 
illuminating  challenges  in  interpreting  the  survey  data,  informing  of  M&S  trends, 
describing  lessons  learned,  and  providing  referrals  to  other  M&S  experts. 
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